An Experimental Multi-Objective Study of the SVM Model Selection problem

Introduction Methods & Materials Results Conclusions Support Vector Machines have been proven to be very effective methods for classification and regression. However, in order to obtain good generalization errors the user needs to choose appropriate values for the involved parameters of the model. SVM model selection problem Tuning the hyperparameters : kernel parameters (γ, degree, coef, etc) regularization parameter C for improving the generalization error. Standard methods (drawbacks): grid search computational complexity (only few parameters) choice of the discretization gradient-based approaches score function must be differentiable; performance depend on the initialization (sub-optimal local minima). Model selection metrics Accuracy: L-fold cross-validation (CV) error. Number of support vectors: in the hard margin case is an upper bound on the expected number of errors made by the leave-one-out procedure. space and time complexity of the SVM classifier scales with it. Algorithms and Libraries Multi-Objective Evolutionary Algorithms (MOEA): Nondominated Sorting Genetic Algorithm II (NSGA-II). LIBSVM library Kernels: RBF: K(u, v) = exp(−γ ∗ |u − v|2) Sigmoid: K(u, v) = tanh(γuT∗ v + coef0) Parameter ranges: log2C ∈ [−5, . . . , 15] log2γ ∈ [−10, . . . , 4] coef0 ∈ [0, 1] Benchmark dataset: Splice dataset: Mean evolution curves for the error and the number of SVs during the optimization using RBF (c) and sigmoid (d) kernels. Support Vector machines (SVMs) are a powerful method for both regression and classification. However, any SVM formulation requires the user to set two or more parameters which govern the training process and such parameters can have a strong effect on the result performance of the engine. Moreover, the design of learning systems is inherently a multi-objective optimization problem. It requires to find a suitable trade-off between at least two conflicting objectives: model complexity and accuracy. In this work the SVM model selection problem is cast as a multi-objective optimization problem, where the error and the number of support vectors of the model define the two objectives. Experimental analysis is presented on a well known test-bed of datasets using two different kernels: RBF and sigmoid. The SVM model selection problem clearly presents the characteristics of a multi-objective optimization problem. The results in this experimental work have shown that it is possible to effectively obtain approximated Pareto fronts of SVM models based on a simple 2-objective formulation, where the accuracy and the complexity of the model are compared for Pareto dominance. Why a multi-objective approach? multiple model generation on a single run; the “best” SVM model can be selected later from the Pareto front according to some higher-level information or preferences; multiple hyperparameters can be tuned at the same time overcoming the limitation of the naive grid-search method; the objective/criteria do not need to be differentiable (as required for the gradient-based methods); efficient exploration of the multimodal search space associated with the parameters. Pareto front of the sampled points using RBF (a) and sigmoid (b) kernel; Parameter surfaces for the diabetes dataset